Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace psf/requests with an unofficial fork #1531

Closed
wants to merge 76 commits into from
Closed

Replace psf/requests with an unofficial fork #1531

wants to merge 76 commits into from

Conversation

Ousret
Copy link

@Ousret Ousret commented Oct 3, 2023 •
edited
Loading

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

  • OS truststore by default, no more certifi!
  • Object-oriented headers. Could bring additional features!
  • Fully type-annotated!
  • HTTP/3 over QUIC.
  • HTTP/2 by default.
  • Exit Python http.client! in favor of h11.
  • Timeout by default.
  • Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
  • All the features you expose are available in all three protocols.
  • Python 3.7+, no sacrifice needed.
  • Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

  • Stricter on emitted requests per RFCs
  • Young project but based on solid bases, knowledge, and experiences.
  • Need to publish new packages to distro. Easy but time-consuming
  • Exit pyopenssl, not supported. more of a pro to me
  • Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

Capture d’écran du 2023-11-27 19-52-12

4.0.0 (unreleased)

  • Switched from the requests library to the compatible niquests. (#1531)
  • Added support for HTTP/2, and HTTP/3 protocols. (#523, #692, #1531)
  • Added support for early (informational) responses. (#752) (#1531)
  • Added support for Happy Eyeballs algorithm via --heb flag (disabled by default). #1599 #1531
  • Added support for IPv4/IPv6 enforcement with -6 and -4. (#94, #1531)
  • Added support for alternative DNS resolvers via --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99, #1531)
  • Added support for binding to a specific network adapter with --interface. (#1422, #1531)
  • Added support for specifying the local port with --local-port. (#1456, #1531)
  • Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495, #1023, #826, #1531)
  • Added support to load the operating system trust store for the peer certificate validation. (#480, #1531)
  • Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
  • Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023, #1531)
  • Added automated resolution of hosts ending with .localhost to the default loopback address. (#1458, #1527)
  • Fixed the case when multiple headers where concatenated in the response output. (#1413, #1531)
  • Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551, #1531)
    This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
  • Improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
  • Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
  • Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)
  • Fixed cookie persistence in HTTPie session when targeting localhost. They were dropped due to the standard library. (#1527)
  • Fixed downloader when trying to fetch compressed content. The process will no longer exit with the "Incomplete download" error. (#1554, #423, #1527)
  • Fixed downloader yielding an incorrect speed when the remote is using Content-Encoding aka. compressed body. (#1554, #423, #1527)
  • Removed support for preserving the original casing of HTTP headers. This comes as a constraint of newer protocols, namely HTTP/2+ that normalize header keys by default. From the HTTPie user perspective, they are "prettified" in the output by default. e.g. x-hello-world is displayed as X-Hello-World.
  • Removed support for pyopenssl. (#1531)
  • Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
  • Removed dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
  • Removed dependency on multidict in favor of an internal one due to often missing pre-built wheels. (#1522, #1531)

Existing plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests.
Future contributions may be made in order to relax the constraints where applicable.

@Ousret Ousret force-pushed the feature-tryout-niquests branch from 46f030c to 16ccc1b Compare October 3, 2023 12:35
@Ousret Ousret force-pushed the feature-tryout-niquests branch from 16ccc1b to 741017e Compare October 11, 2023 18:44
@Ousret

This comment was marked as resolved.

Sign in to view
Ousret
Ousret commented Oct 11, 2023
View reviewed changes
tests/test_json.py Outdated Show resolved Hide resolved
@Ousret

This comment was marked as resolved.

Sign in to view
@Ousret Ousret force-pushed the feature-tryout-niquests branch 3 times, most recently from faf9aef to c12f000 Compare October 13, 2023 16:37
@codecov-commenter
Copy link

codecov-commenter commented Oct 13, 2023 •
edited
Loading

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (4d7d6b6) to head (7cd6579).
Report is 370 commits behind head on master.

❗ Current head 7cd6579 differs from pull request most recent head 55aea9c

Please upload reports for the commit 55aea9c to get more accurate results.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #1531       +/-   ##
==========================================
- Coverage   97.28%       0   -97.29%     
==========================================
  Files          67       0       -67     
  Lines        4235       0     -4235     
==========================================
- Hits         4120       0     -4120     
+ Misses        115       0      -115

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Ousret

This comment was marked as outdated.

Sign in to view
@Ousret

This comment was marked as resolved.

Sign in to view
@Ousret Ousret force-pushed the feature-tryout-niquests branch 4 times, most recently from 52653e4 to 247b382 Compare October 16, 2023 05:45
@Ousret Ousret force-pushed the feature-tryout-niquests branch 2 times, most recently from 0b4eebb to 7dd1152 Compare October 23, 2023 10:56
@Ousret

This comment was marked as spam.

Sign in to view
@Ousret

This comment was marked as spam.

Sign in to view
@Ousret Ousret force-pushed the feature-tryout-niquests branch from 7dd1152 to 7257a74 Compare November 16, 2023 06:45
@Ousret

This comment was marked as spam.

Sign in to view
@Ousret Ousret changed the title ⚗️ Try compatible fork Niquests to supercharge HTTPie ⚗️ Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info Nov 16, 2023
@Ousret Ousret force-pushed the feature-tryout-niquests branch from 7257a74 to 4c246e4 Compare November 27, 2023 18:59
@Ousret

This comment was marked as spam.

Sign in to view
@Ousret Ousret force-pushed the feature-tryout-niquests branch from 4c246e4 to f034379 Compare November 27, 2023 19:06
@Ousret Ousret changed the title ⚗️ Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info Dec 10, 2023
@dwt
Copy link

dwt commented Dec 20, 2023

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

@Ousret Ousret force-pushed the feature-tryout-niquests branch from f034379 to df42bba Compare January 1, 2024 18:22
@Ousret

This comment was marked as spam.

Sign in to view
@Ousret Ousret changed the title Add support for HTTP/2, HTTP/3, System CA, OCSP Revocation, Extended Conn Info Proposal v4.0 b1 Jan 1, 2024
@dwt
Copy link

dwt commented Oct 15, 2024

@jkbrzt Aren't you the maintainer here? Could you give an update on this?

@Ousret

This comment was marked as spam.

Sign in to view
@techknowlogick
Copy link

@Ousret, like the others in the thread, I also wanted to thank you for your work on this PR. I have been building it with my NixOS flake https://gitea.com/techknowlogick/tklk-flake as I run the PR version daily.

@Ousret Ousret force-pushed the feature-tryout-niquests branch 2 times, most recently from bb4ce6e to bbdb20c Compare October 22, 2024 16:32
@Ousret Ousret force-pushed the feature-tryout-niquests branch from bbdb20c to e375c25 Compare October 22, 2024 16:33
@markstos
Copy link

@Ousret At what point do you go ahead and publish your work as a fork?

@Ousret

This comment was marked as spam.

Sign in to view
@Ousret Ousret closed this Oct 28, 2024
@markstos
Copy link

@Ousret Thank you for all your work on this.

As a httpie user, I'm also interested in "correctness over performance". I would rather be able to test HTTP/2 and HTTP/3 connections a fraction of a second slower than to not be able to test them at all.

I arrived at this pull request because I was affected by one of the amount two dozen issues that it closes. If the original maintainer doesn't merge it, I hope it's packaged as a fork. Though, my own open source plate is also full as well, and I'm not volunteering at the moment, either.

@jkbrzt
Copy link
Member

jkbrzt commented Oct 28, 2024 •
edited
Loading

Sorry to see this. I expressed concerns over performance as with Niquests, the most common use case of fetching a JSON response is 50% slower (including over HTTP/1). My last words were that I'll do more testing. For the record, we have been in touch over email (as well as sponsoring Ahmed) for over half a year. I wish all the best to both Ahmed and his project.

@httpie httpie locked and limited conversation to collaborators Oct 28, 2024
@Ousret Ousret deleted the feature-tryout-niquests branch October 29, 2024 14:28
@jkbrzt jkbrzt changed the title Proposal 4.0 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...) Replace psf/requests with an unofficial fork Oct 30, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@janbrasna janbrasna janbrasna requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
10 participants
@Ousret @codecov-commenter @dwt @jkbrzt @mshahat @markstos @ducaale @MattElek @techknowlogick @janbrasna